Planning To Fail, Not Failing To Plan: Risk-Taking And Recovery In Task-Oriented Dialogue

We hypothesise tha t agents who engage in taskcontains 128 such dialogues; in this work we examined oriented dialogue usually t ry to complete the task with eight plus a set of dialogues from the pilot s tudy used the least effort which will produce a satisfactory soin Shadbol t ' s work [17]. Agents who wish to avoid plan lution. Our analysis of a corpus of map navigation task dialogues shows tha t there are a number of different aspects of dialogue for which agents can choose either to expend extra effort when they produce their initial utterances, or to take the risk tha t they will have to recover from a failure in the dialogue. Some of these decisions and the strategies which agents use to recover from failures due to high risk choices are simulated in the JAM system. The human agents of the corpus purposely risk failure because this is generally the most efficient behaviour. Incorporat ing the same behaviour in the JAM system produces dialogue with more "natural" s t ructure than tha t of t radit ional dialogue systems. I n t r o d u c t i o n There are a great number of different dialogue styles which people use even in very restricted task.oriented domains. Agents can choose different levels of specificity for referring expressions, ways of organising descriptions, amounts of feedback, complexities of explanation, and so on. This work first identifies a number of aspects of task-oriented dialogue along which agents can make choices and identifies these choices in terms of how much effort the agent must expend in order to generate ut terances in line with them. In general, expending more effort in building an explanation means tha t the explalnee is more likely to unders tand it as is; thus we can classify some choices as being "higher risk" than those which take more effort to generate but which are more likely to succeed on the first a t tempt . Then it identifies a number of recovery strategies which agents use when risky hehaviour has led to a failure in the dialogue. The choices which agents make show n trade-off of when effort is expended in the dialogue; agents can either expend effort early in order to head off later difficulty, or take the risk of having to expend more effort in an a t tempt at recovery. For instance, consider the domain, first described in [5], in which two part icipants who are separated by a part i t ion have slightly different versions of a simple map with approximately fifteen gross features on it. The maps may have different features or have some of the features in different locations. In addition, one agent has a route drawn on the map. The task is for the second agent to *This research was supported by a postgraduate studentship from the M~trshMl Aid Commemoration Commission and supervised by Chris Mellish. The author's current address is HCRC, 2 Buccleuch Place, University of Edinburgh, Edinburgh Ett8 9LW, Scotland. failure may structure their explanations carefully and elicit feedback often, hehaving similarly to agent A in Shadbol t ' s example 6.16: A: have you got wee palm trees aye? B: uhu A: right go just + a wee bit along to them have you got a swamp? B: er A: r ight well jus t go + have yon got a waterfall? On the other hand, agents who are willing to rely on interruptions from the par tner and recovery from failure might behave more like agent A in Shadbolt ' s example 6.11: A: and then + go up about and over the bridge B: I 've not got a bridge I 've got a lion's den and a wood A: have you got a river? Either of these approaches is likely to bring the agents to successful completion of the task. However, it is also possible to include too little information in the dialogue, as in the following case, Shadhol t ' s example 6.21: A: right + you 're going to have to c ro~ the river B: how? A: dinnae ken + any way you want... It is equally possible to give too much information, as in Shadbolt's example 6.27: B: ah right + erm + oh yes + er + I have a crashed plane marked here + can I + check this + my crashed plane is ABOVE + it 's in the BASE of the quadrant + top r ight hand imaginary quadran t of the + erm + picture + yes er + tha t SOUNDS too high for me + A: er In this case, B provides so much information tha t A is unable to process it, and they eventually abandon this section of the dialogue. This work looks at the differences between the approaches which human agents use to complete the map task and simulates them using the JAM system. Understanding and compar ing the different h u m a n approaches to task-oriented dialogue can help us to create more robust computer dialogue agents. C o m m u n i c a t i v e P o s t u r e Our work extends Shadbol t ' s analysis of the map task da ta [17]. He identifies a number of "communicative posture parameters" or aspects of the dialogue for AcrEs DE COLING-92, NANTES, 23-28 AO~ 1992 8 9 6 PROC. OP COLING-92, NANTES, AUG. 23-28. 1992 which an agent may make tit(-" choice of how to proceed, and classifies the possible settings in ternm of risk: for the most part , high risk settings leave the par tner to infer information aml risk the possibility of plan failure, while low risk settings are more likely to work as planned, lie then argues that hmnan agents decide upon their communicative postures according to the Principle of Parsimony, which is "a behavioural princio pie which instructs processors to do no more processing than is necessary to achieve a goal." (pg. 342) Agents choose the settings for each individual parameter which they believe will prove most etlicient. Shadbolt identifies seven different communicative posture parameters. Our own analysis extends tds by clearly separat ing out aspects of being a hearer from those of being a listener mt~l hy making the behaviour o f the parameters more in~ dcl)endcnt of each other mid subsequently dividing them into sets dcpcnding on which par t of an agent 's planning they affect. Wc divide ut terance planning into diffcrcnt stages similar to Appel t ' s [4] for this part of the analysis. The following revised set i)rovides a more solid foun.. dation on which to build the implementation found in the JAM system: T a s k P l a n n i n g P a r a m e t e r s '.l'hese parameters affect which task plan an agent chooses. In the map domain, task plans determine the choice of descriptions for sections of the route and for the location of objects. O n t o l o g y : Thc choice of concepts to use when buildins an explanation, l l igh risk agents construct simple and short descriptions, providing ms littlc i n fo f lnation as they think the par tner will allow, willie low risk agents provide precise, detailed explauatioas even if tha t involves using fairly COml)lex background concepts and introducing new concepts into the dialogue. O n t o l o g i c a l l l~esolution: The choice of concepts to ask about when hearing an explanation, l[igh risk agents asccpt the level of detail which is off~red to them, wbile low risk ones ask how concepts are related if they think tha t the relationship may be an important piece of background for tim explanation. P a r t n e r M o d e l l i n g : Wltether or not to heed a model of the par tner while building an explanation. High risk agents do not, while low risk agents do, tailoring tile explanation f(u" the partner . It takes more effort in the first instance to buihl an explanation which is tailored to the partner , but the explanation is more likely to succeed without revisions. Ontology and par tner modelling are implemented in tile J AM system by means of an evaluation selmme for possible task plans which rates descriptions differently (let)ca(ling on whether these parmneters are set to low or high risk. Low risk ontology prefers descriptions which rcfer to many map objects over simpler ones; if there arc sevcral descriptions of equal complexity, low risk l)artocr modelling prefers descril)tions which (Io not refer to map objects tha t may I)c unknown to the partner. Ontological r(~)luLion is not irnp]enlente(l in tile JAM system because JAM agents are not capable of the spatial reasoning required to determine what other map objects are relevaat to a given description. D i s c o u r s e P l a n n i n g P a r a m e t e r s These parameters affect the s t ructure of the dim course, given the information from the task plan which numt be conveyed. D i f f e r ence : Whether or not agents assume tha t their modeln of the domain are the stone unless proven otherwise. High risk agents make this assumption, while low risk agents do not, making them precede new concepts in the dialogam with subdialogues which establish certain knowledge of the par tner ' s knowledge such as direct questions about the s ta tus of the concepts. A low risk difference sett ing makes the (lia~ logue longer and hence requires more effort, but ales provides a greater s t rength of evidence about the par tner ' s beliefs [7] than does relying on the par t her 's feedback to the explanation itself. This parameter is implemented in the JAM system by means of optional prerequisites on discourse plans which introduce new concepts; low risk agents expand the prerequisites, while high risk agents do not. C o h e r e n c e : Whether or not the agents orgauisc their diseourse coherently, lligh risk agents produce utterances in whatever ordcr they think of thcm, whereas low risk agents t ry to order them in some way which will make the discourse e ~ i c r for the partner . This parameter is not implenmnted in the JAM system because, map task par t ic ipants do not often organise the discourse except as if they were physically following the route. In less well s t ructured domains, it could be implemented using, for instance, RST [11] or focus trees [12]. U t t e r a n c e R e a l l s a t i o n P a r a m e t e r s Thrum parameters affect the way in which each ut terance in the given discourse s t ructure is realised or understood. C o n t e x t A r t i e u l a t l o n : Whether or not the agents signal awkward context shifts, llere context is loosely dcfincd as tile goM which is supported by the current par t of the dialogue; in the map task, contexts carl either be goals of sharing knowledge about a section of tile route or tim location of an object, t l igh risk agents do not signal awkward context shifts, while low risk agents use mcta~comments, changes in diction, or sot[m other means to mark tile new context. A limited version of the low risk sett ing is imphmmnted in JAM which introduces a reels-comment into the dialogue whenevcr a context shift occurs. C o n t e x t ILesoluf ion: Whether or not agents ask for clarification of awkward context shifts. Low risk agents ask tile par tner what the current context is or make their assumptions clear when they are unsure, whereas high risk agents simply choose the most likely context. This parameter is not implemented in the JAM system because JAM agents use a language which (tt~s not allow for ambiguity of context. ~ 'ocus A r t i c u l a t i o n : Wllether or not agents signal awkward focus shifts, liere, focus is defined specifically for tile map task in terms of distance on the ACYES DE COl,INGo92, NANrEs. 23 28 AO~"H 1992 8 9 7 Ptto(:, OF COI,IN(I-92, NANrES. AUU. 23-28, 1992 map and semantic relationships among map iea~ tures. Low risk agents use meta-comments or moditiers on referring expressions to signal awkward focus changes, and high risk agents do not. Focus articula~ tion is not implemented in the ][AM system because JAM agents are not capable of the spatial or semantic reasoning required to calculate focus; given these abilities, low risk agents could use some theory of how focus usually moves (such as tha t of Grosz and Sidner [9]) to determine whether or not signaling a part icular shift is necessary. F o c u s R e s o l u t i o n : Whether or not agents ask for clarification of awkward focus shifts. Low risk agents ask the par tner what the current focus is or mark their assumptions in some other way, whereas high risk agents simply choose the most likely focus. Low risk focus resolution could be implemented by haying low risk agents ask for clarification whenever a focus shift does not conform to sonm theory of focus, with high risk agents "guessing" the current focus. Spec i f i c a t i on : Whether or not agents construct referring expressions carcfidty. Low risk agents generate referring expressions which are roughly minimally unique, whereas ifigh risk agents generate whatever expression comes to mind, even if tha t expression is underor over-specific. This parameter could be implemented in the JAM system nsing, for instance, work by Dale [8] and Reiter [16]. D e s c r i p t i o n R e s o l u t i o n : Whether or not agents decode referring expressions carefully. Low risk agents ask for clarification of ambiguous referring expressions, while high risk agents simply choose the mostly likely referent. This parameter could have an implementation similar to tha t of the specification parameter , but from the point of view of the addressee. M e t a P l a n n i n g P a r a m e t e r This parameter affects an agent 's choice of how to continue from the current si tuation in a dialogue. P l a n C o m m i t m e n t : Whether or not agents decide to replan easily. Low risk agents tend to stick to the current plan unless there is sufficient proof tha t tile new plan is better, whereas high risk agents of_ ten replan when they encounter failures even without carefully checking the viability of the new plan. Frequent changes in plans are likely to confuse the par tner and lead to difficulty in the dialogue, especially if the agent 's context ~t icula t ion setting is also high risk. This parameter is implemented in the JAM system by means of a "replanning threshold" which is added to tile estimated cost of a replan and which makes replanning seem less efficient to low risk agents that to high risk ones. Of course, the choice is not between extremes, but among points on a spectrum which generally reflects the amount of effort to be expended. Shadbolt adapts the Principle of Pars imony to state tha t agents make the choices which they believe will lead to the lowest effort solution for the entire task. In each case, high risk agents may lose the efficiency advantage which they gained by using less effort initially, if their plans fail and they have to expend more effort to recover from the failures. Recovery strategies are more often needed by high risk agents than by low risk ones. R e c o v e r y S t r a t e g i e s Our analysis has uncovered the following recovery strategies. Some strategies are only first steps towards finding a solution for the failure, mid one, goal adoption, is also useful in other circumstances. We use the same basic definitions for repair and replanuing as in Moore's work [13]. G o a l A d o p t i o n : The agent may infer the par tner ' s goals from sonm par t of the dialogue he or she has initiated and adopt them as his or her own. C e d i n g t h e T u r n : The agent may simply not take any action and hope tha t his or her inaction will force the par tner into initiating the recovery. E l a b o r a t i o n : If an explanation has not been given in enough detail, the explainer may fill in the gaps. Omis s ion : If an explanation has been given in too nmch detail, the agents may agree to discard some of the information. This is especially useful in the map task if some description of the route or of the location of an object OIL the map turns out to hold for one version of the map but not the other. R e p e t i t i o n : Under any circumstances, an agent may simply repeat whatever action has already failed ill the hopes tha t it will work the subsequent tinve. I g n o r i n g t h e P rob l ean : An agent may ignore a problem and hope tha t it will disappear. R e p a i r : If a plan has failed, then checking each of the prerequisites of the plan ill turn to see if they are satisfied may lead to a diagnosis. In the map task, plan prerequisites have to do with knowledge about objects on the map. A plan will fail if an agent presupposes tha t the par tner has knowledge which he or stle does not have. Since the knowledge transferred in the map domain is so simple, it is sufficient in a repair to re-execute any failed prerequisites, even if the plan has already been completely executed. R e p l a n n h i g : If a plan has failed, then an agent may a t tempt an entirely different plan with the same effect. In the map task, this involves using a different description for the information under consideration or t rying a different approach altogether. There are many past systems which have incorporated some form of recovery from plan failure (e.g., [2], [19], [14]). However, very little work has been done on incorporat ing more than one recovery s t ra tegy into tile same system. Moore's [13] work allows the use of repair, reinstantiat ion, aud replanning, but uses a strict ordering on these strategies to determine which one to try next. Moore's system first a t t empts any poesible repairs, then any reinstantiat ions, and then, only as a last resort, replanning. Neither Moore's ordering nor any other can account for the variety of behaviours which is present in the human map task corpus. In addition, Moore's system only considers replanning when there has been a plan failure, whereas human agents sometimes switch plans when they flesh out enough of the details and discover tha t the plans which they have adopted are leas efficient than they had expected. The solution to these shortcomings is to invoke the Principle of Pars imony and to allow agents at every choice point to decide what to do next based on an estimates ACIT~ DE COLING-92, NANaa~s, 23-28 hOt'It 1992 8 9 8 PRoc. OV COLING-92, NANTES. AUG. 23-28, 1992 Figure 1: The Structure of a JAM Agent's Planner